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We consider a discrete-time Markov chain (X*, Y"'), t = 0,1,2, .. ., 
where the X-component forms a Markov chain itself. Assume that 
(X') is Harris-ergodic and consider an auxihary Markov chain {K'} 
whose transition probabihties are the averages of transition probabil- 
ities of the F-component of the {X, y)-chain, where the averaging is 
weighted by the stationary distribution of the X-component. 

We first provide natural conditions in terms of test functions en- 
suring that the K-chain is positive recurrent and then prove that 
these conditions are also sufficient for positive recurrence of the orig- 
inal chain (X*,y*). The we prove a "multi-dimensional" extension 
of the result obtained. In the second part of the paper, we apply our 
results to two versions of a multi-access wireless model governed by 
two randomised protocols. 



1. Introduction. We develop an approach to the stabihty analysis 
based on an averaging Lyapunov criterion. More precisely, we consider a 
discrete-time Markov chain {Z^,t = 0,1,2,...} with values in a general 
state space which has two components, Z* = {X^,Y^). We assume that the 
first component {X*} forms a Markov chain itself. We further assume that 
the Markov chain {X*} is Harris-ergodic, i.e. there exists a unique station- 
ary distribution irx and, for any initial value = x, the distribution of 
converges to the stationary distribution in the total variation norm: 

sup |P(X* G A) - 7rx(A)| ^ as t ^ oo. 

In many known models in applied probability, the time evolution of the 
process describing the state of the system may be expressed as a multi- 
component Markov chain where one of the components is either a Markov 
chains itself (see, e.g. [13], [14], [17], [7] and references therein) or, more 
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generally, behaves as "almost stable", changing only relatively slowly, while 
the other component changes more rapidly (see, e.g., [22]). 

Our aim is to formulate and prove a stability criterion for the two-component 
Markov chain = (X*,y*) by making links to an auxiliary Markov chain 
where {X^} is an i.i.d. sequence with common distribution ttx- 

First, one can recall a standard approach to the stability analysis which 
may be used for this model and which is based on the following two-step 
scheme. To simplify the explanation, assume that the first-component Markov 
chain {X*} is regenerative with regeneration times < Tq < Ti < . . ., so 
that all values X'^" are equal to, say, x*^ = const. 

Step 1. Consider an embedded 2-component Markov chain = 
(X"^" , y^" ) at the regenerative epochs Tq < Ti < . . . < < . . . . Since 
X" = , the sequence {Y^} also forms a Markov chain. For this chain one 
may show that, under appropriate assumptions and for an appropriate test 
(Lyapunov) function L, the drift EL(y^ | = y) — L{y) is bounded from 
above by the same constant for all values of y and is also uniformly negative 
if y is outside of a certain "bounded" set, and hence this bounded set is 
positive recurrent. For that, it is necessary to find or estimate from above 
the average drift of L(y") over a typical cycle, say Ti — Tq. By introducing 
further smoothness condition(s), one can then ensure that the Markov chain 
{y"} is also Harris ergodic. 

Step 2. According to step 1, one can say that the Markov chain {(^", Y"^)} 
is regenerative, then verify its aperiodicity and conclude that it is Harris 
ergodic. 

However, the first step of the proposed scheme may not be implementable 
if the transition function is not sufficiently smooth (and, in particular, has 
discontinuities) as then it may be difficult to find/estimate the value of the 
average drift of L{Y) during a typical cycle. 

Another approach to proving stability is the use of fluid limits (see, e.g., 
[9], [10] and Chapter 10 of [18]). To apply this method, one needs to prove the 
convergence of an appropriately scaled version of the process, establish the 
dynamics of the limit, show that these dynamics are in some sense stable and 
then prove that this "stability" of the limit implies stability of the original 
Markov Chain. This is however often rather cumbersome and technically 
involved and moreover requires additional assumptions on the Markov chain. 

In this paper we introduce a different approach which is based on the 
following idea. 

First, we find the (unique) stationary distribution ttx for the X-component. 
Second, we introduce an auxiliary (time-homogeneous) Markov chain {y*} 
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with transition probabilities 

(1) P(y*+^ e . |y* = y) = f TTxidx)P{Y^ e ■ \ = x,Y° = y) a.s. 

Jx 

This Markov chain can also be viewed as an outcome of the following recur- 
sive construction of a two-dimensional Markov chain (X*,y*) whose first 
components have common distribution ttx- First, for each t, we de- 
termine as a random variable which does not depend on all r.v.'s 
{X^}k<t, {y^ykKt, and has distribution ttx- Second, we determine as 
a random variable which is 

(a) conditionally independent of {X''}k<t-, {^^}k<t~i given and Y^, 
and 

(b) has distribution 

P(y*+^ G • = X, y* = y) = P(y^ G • \ X^ = x,Y^ = y) a.s. 

Then, indeed, {Y^} forms a (time-homogeneous) Markov chain with transi- 
tion probabilities given by (1). 

In this paper, we formulate and prove general stability criteria in terms 
of averaging Lyapunov functions. In particular, we introduce natural con- 
ditions for the following implication to hold: if the Markov chain {X^} is 
Harris ergodic then the Markov chain {Y*} is positive recurrent. The same 
conditions are then proved to imply that the Markov chain {{X^ ,Y^)} is 
positive recurrent too. 

We formulate and prove general stability results (Theorems 1 and 2) in 
Section 2. In Theorem 1 we obtain a "one-dimensional" result and then in 
Theorem 2 its multivariate analogue. Then in Section 3 we study the sta- 
bility of two systems with multiple access random protocols in a changing 
environment. The proofs for their stability are carried out by applying The- 
orem 2 and using the monotonicity arguments. In these particular scenarios 
we also show that the stability conditions are necessary (if they do not hold, 
then the system under consideration is unstable). 

2. Stability of two-component Markov chains using averaging 
Lyapunov functions. In this section we consider a general framework 
for the stability of a Markov Chain containing two components one of which 
is a Markov Chain itself. 

In what follows, we write for short Pxi- ■ •) instead of P(. . . | Xq = x), 
Py{- ■ •) instead of P(. . . \ Yq = y), and Px,y{- ■ •) instead of P(. . . | Xq = 
x,Yq = y). We use similar notation E^^, Ey, and ^x,y for conditional expec- 
tations. 
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2.1. One- dimensional case. Let {X^} and {Y*} be random sequences 
taking values in measurable spaces {A!,Bx) and {y,By), respectively, with 
countably generated sigma-algebras Bx and By- Assume that {(X*,y*)} 
is a Markov Chain on the state space {X x 3^, Bx x By) (here Bx x By is 
the minimal sigma-algebra generated by sets Bi x B2 where Bi € Bx and 
-B2 £ ^y)- Assume also the following: 

^0. {^*} is a Markov Chain with "autonomous" dynamics: for all x G 
X and y (z y, 

Px,y{^' e •) = Px(^' e •)• 

Further, the Markov chain {X^} satisfies the following conditions: 
Al. It is aperiodic; 

A2. There exists a set V S Bx such that 

(2) r = t{V) = mm{t > I : X^ £ V} < 00 - a.s. 
for any initial value Xq = x £ X and, moreover, 

(3) So = sup 'EixT < 00. 

(We say that the set V is positive recurrent for {X^} if conditions (2) and (3) 
hold.) 

A3. The set V admits a minorant measure, or is petite (in the termi- 
nology of [19]), i.e. there exist a number < p < 1, a measure and an 
integer m > 1 such that 

Px(X'"e-)>pM-) 

for any x (zV. 

Conditions Al~A3 imply that the X-chain is Harris ergodic, so there 
exists a stationary distribution vr = ttx such that 

(4) sup |P^ (X* eB) - tt{B)\ 

as t ^ 00, for any x £ X. Moreover, Conditions A1-A3 imply that 

(5) convergence in (4) is uniform in x G V, 

see e.g. [6], [23] or [16]. We can formulate also a coupling version of conditions 
(4)-(5): 

for any x £V, there is a coupling of {X^} and of a stationary Markov chain 
{X*} having distribution vr such that 

(6) u = mm{t : X'' = X^ , for all k>t] < 00 - a.s. 
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for Xq = X and 

(7) 5t := sup Vxiv > t) 0, t oo, 

x&V 

see Appendix A for a proof of (6) and (7). 
Introduce a function 



(8) Li(x) 



E^T, if x^V, 
0, if a; G V. 



Then 

(9) E,.(Li(Xi)-Li(x)) =-1 
for all X ^ y and 

(10) E^. (Li(X^) -Li(x)) < supE^r- 1 < oo 

x<^V 

for all X (z V. These inequalities follow from observing that txi = — 1 
Px — a.s. if Xi ^ V. Inequalities (9) and (10) mean that the function Li is 
an appropriate Lyapunov function for the Markov Chain {X*} in the sense 
that it satisfies the standard conditions for the Foster criterion to hold. 
It is known (see, e.g., [6], [23] or [16]) that Conditions A1-A3 imply that 

(11) lim - sup E^Li(X*) = 0. 

B. For the sequence we assume that there exists a non-negative 
measurable function L2 such that: 

Bl. The expectations of the absolute values of the increments of the 
sequence {L2(5^*)} are bounded from above by a constant U: 



sup E,,^|L2(y^) -L2(y°)| < f/< 



00. 



B2. There exist a non-negative and non-increasing function h{N), N > 
such that h{N) J, as A'" ^ 00, and a measurable function f : X ^ 
(—00, 00) such that 

f{x)7T{dx) := -e <0 



X 



and 

(12) E,,j, (L2 (y^) - L2(2/)) < fix) + /i(L2(2/)) 
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for all X (z X and y y. 

It follows from Condition Bl that, without loss of generahty, the function 
/ may be assumed to be bounded: 



(13) 



sup |/(x)| = K < oo. 



Remark 1. Note that conditions B are sufficient for stability of the 
chain Y introduced in (1), due to the standard Foster criterion. We are 
going to show that they ( together with conditions A ) are also sufficient for 
stability of the original chain {Z^}. 

Lemma 1. Conditions A and B imply that, for some positive integer to 
and for any integer t > to there exists a positive number Nq = No^to) such 
that if L2{y) > A'^o, then 

(14) E,,y (L2(y*) - L2(y)) < -tA 

for all X ^ V with A = e/10. 

Proof. Due to conditions (5) and (13), for any ci € (0, 1) one can choose 
a number ng such that 



(15) 



sup sup 

t>no x&V 



E4/(X*)) - / f{z)^{dz) 



<Ci. 



Let t = no + m. Then, for x € F, ^/ S 3^ and for U from Condition Bl, 

t-1 



E,,,(L2(>^*)-^2(y)) 



= EE.,.(^2(y^+^)-L2(yO) 

i=0 

t-l 

< Uno+Y, Ex,, (E (L2 (r+^) I X\Y') - {Y')) 

i=no 
t-1 

< Uno+J2B,,y{f{X')+h{Y')) 

i=no 

t-1 

< Uno -{e- ci)m + ^ B^^yh {Y') 

i=no 

e-ci- h{0)I'^^y ^nijnL2(y*) < iV^ - h{N) 



< Uno — m 
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where N is any positive number. Take ci = e/5 and then niQ = max(no, 5C/no/e) 
and to = ^0 + ^o- Then, for any m > rriQ, let N be such that h{N) < e/5 
and /i(0)(?io + mfU/N < e/5. Further, let Nq = 2N. If L2{y) > Nq, then 



p 



min L2(Y') < N] < tsupP,,„ (l2{Y^) - L2{u) < -N/t 

l<«<t / z,u V 

< h{0)t^U/N < e/5, 

where we applied the Markov inequality. Hence, for x € F and y such that 
L2{y) > No, 

E.,,(L2(y*)-L2(y))<-m|<-t^ 
and therefore inequality (14) holds with A = e/10. □ 

Theorem 1. Under the conditions A and B there exists Nq such that 
the set D := V X {y : L2{y) < Nq} is positive recurrent for the Markov 
Chain {(X\Y^)}. 

Corollary 1. Assume Conditions A and Condition Bl to hold. As- 
sume further that, uniformly inx £ the conditional distribution Px,y{L2 [Y^] 
-^2(2/) € ■) converges weakly and in L\ to a limiting one, say Hx{-), as 
L2{v) — > 00, where 

Tr{dx)yHx{dy) 




IX Jy 

is negative. Then the Markov Chain {(X*,y*)} is positive recurrent. 

Proof of Theorem 1. We use the following extension of the Foster 
criterion (see, e.g., [12, Theorem 1] or [8] and references therein). Let L : 
X X y [0,00) be a measurable function. Then a set D = {{x,y) : 
L{x, y) < N} is positive recurrent if there exists a positive integer-valued 
measurable function T on X x y such that 

T{x,y) ^ ^ 
x,y max(l,L(x,y)) 

sup Ex,yL ('x^(^.y)^yr{^,J/)'j < 00 

ix,y)<=D ' ^ ' 

and 

(16) E,,,L (x^{-^v\y'^^-^vA - L{x,y) < -cT{x,y) 



for some c > and for all {x, y) ^ D. 
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Let H > U with U from condition Bl and let to be such that 
(17) sup)supE,Li(X*) < 

t>to I x€V 

which is possible due to (11). Here again A = e/10. Let nQ,m and be 
chosen according to Lemma 1 with to from (17). Let again Nq = 27V. 
We take the set D = V x {y : L2{y) < A'o} and define a function 

L{x,y) = HLi{x) + L2{y). 

Further, take T{x, y) = 1 if either {x,y) & D or x ^ V and T(x, y) = t if 
X and -^2(2/) > -^Oi where t = no + m is from the proof of Lemma 1. 
Now, if {x,y) € D, then 

E,^yL{X\Y^)-L{x,y) <H sup E,T + U. 

x€V 

li X ^V, then 

E^,yLiX^,Y'^) - L{x, y) < -H + U <0. 
Finally, if x eV and L2(y) > iVo, then, by (14), 

□ 

2.2. Multivariate case. Now we formulate and prove a multivariate ana- 
logue of Theorem 1. We modify the model as follows. We continue to assume 
that {(X*, y*)} is a Markov Chain on the state space (<Y x y, Bx x By) and 
that {X*} is a Markov chain satisfying conditions A0~A3. But now we as- 
sume that the state space 3^ is a product of M spaces 3^ = x . . . x yM 
(with the product sigma-algebra), so the y-component of the Markov chain 
has M coordinates, = (Y^, . . . ,Y^,j). Clearly, the results of the previous 
section hold in this case too. However, they turn out not to be applicable 
in the examples we are going to consider in the second part of the paper, 
and in this section we develop conditions only involving each individual co- 
ordinate of the y-chain. We believe that these conditions may be useful in 
many applied models and, in particular, where the y-chain has a number of 
coordinates which may be dependent but have simple individual dynamics. 

We assume that there is a non-negative function L2,j defined on J^j and 
that conditions similar to i?l — B2 hold for each coordinate. 
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Bl. For any i = 1, . . . , M, the expectations of the absolute values of the 
increments of the sequence {L2^i{Y^)} are bounded from above by a constant 
U: 

sup E,,^ \L2,i (y/) - L2,^ \<U< 



oo. 



B2. For each i, there exists a function hi{N),N > such that hi{N) J, 
as — 7- oo and a measurable function /j : X — )> (—00,00) such that 

SUPa; |/i(2;)| := -f^i < 00, 



(18) / h{x)^{dx) :-- 



-Si < 



and 

(19) E^^y {L2,i (y/) - L2,^{y^)) < fiix) + h^{L2,i{yi)) 

for all X G and y y, where the RHS depends only on the x and on the 
zth coordinate of the y. 

In the multivariate case, the statement of Lemma 1 holds for each coor- 
dinate i, and we restate it here for convenience. 

Lemma 2. Conditions A and B1-B2 imply that, for some integer t' and 
for any integer t > t' , there exists a positive number Nq such that, for any 
i, if L2^i{y) > Nq then for all x (zV 

(20) E^^y (L2,.(y*) - L2,.(y)) < -tA, 
where A = minSj/lO. 

If the sequence {(X^, .., X^, 1^*)} is a Markov chain for each i, then 
clearly. Theorem 1 also holds for each coordinate i, meaning that the Markov 
chain {(X^, ..,X^,j,Yf')} for any coordinate i is positive recurrent. However, 
this does not imply the positive recurrence of the entire chain 
{{Xl, ..,X'^.j,Y^, ..,Y^j)}. In order to obtain conditions for stability in the 
multivariate case (also without assuming that each of the above sequences 
is itself a Markov chain), we require an extra assumption to hold: 

B3. For each i it holds that 

E.,, {L2Ay') - L2Ay)\Y' = y) ^ 

sup sup > 

t>Nyey,x£V i 



as N ^ 00. 
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Assumption B3 means that the drift of the function L2,j in t steps grows 
slower than any hnear^imction of t. Below we provide a simple condition B 
which is sufficient for B3 to hold. Note that condition B is a stronger version 
of condition B2 and that it can be easily verified in many applications. 

Lemma 3. Assume that the following condition holds: 
B. For each i, there exist a non-negative and non- increasing function 
hi{N),N > such that hi{N) as N ^ 00, and a family of mutually 
independent random variables {(p\. j}, x € , t = 0, 1, . . . such that, 

(i) for each t and i these random variables are uniformly integrable; 
(a) for each i and x, the random variables {f^. i, t = 0,1, . . .} are identi- 
cally distributed with common distribution function F^^i, which is such 
that Fx^i{y) is measurable as a function of x for any fixed y; 
(Hi) the inequality 

(21) L2,i - L2,i {Yl) < + /i.(L2,.(y/)) a.s. 

holds for all x £ X , y £ y and t = 0,1, . . . ; 
(iv) functions fi{x) = E(^^ ■ satisfy condition (18). 

Then assumption B3 holds too. 

We will now formulate the main theorem and then prove it. A proof of 
Lemma 3 is presented after the proof of the main theorem. 

Theorem 2. Under assumptions A and assumptions Bl — B3 (or as- 
sumptions Bl and B), there exists Ni > Nq such that the set D := V x {y : 

M 

I-'2{yi) A^i} is positive recurrent for the Markov Chain {(X*,y*)}. 

l=i 

Proof of Theorem 2. We may apply Lemma 2 to each of Yi and may 
therefore assume that inequality (20) holds for each coordinate, with the 
same t' and the same A. 

Due to condition (11), we may also assume that t' is such that 

(22) supySupE,.Li(X*) < A, 

where H is any positive number larger than MU with U from assumption 
Bl. 

Choose also no such that 

(23) E,,, [L2^^{Y') - L2,.(y)|F° = v) < 
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for all t > riQ and for all y. This is possible due to condition B3. 

We again use Theorem 1 of [12]. Take A^i = MNq so that D = V x {y : 

M 

^^iVi) < MNq} and introduce test function 

L{x, y) = HLi{x) + ^ L2,i{yi). 

i 

Let T{x, y) = 1 if either {x,y) G D or x ^ V and T{x, y) = ti := max{t', no}, 
otherwise. 

Now, if (x, y) G D, then 

E^,yL(X\ y^) - L(x, y)<H sup E^r + MU < oo. 
If X ^ y, then 

E3;,yL(X\ y^) - L(2;, y) <-H + MU < 0. 
And if X £ V and ^ L2^i{yi) > MNq, then L2^i{yi) > Nq for at least one 

i=l 

index i. Denote by k the number of such indices. Then, by (20), (22) and 

(23) , 

E,,,L (x^(-'^),y^(-'^))-L(x,y) < Hti^-khA+{M-k)^h < -h^ 

as k > 1. Clearly, sup^ ,^ max(i^L(x y)) ^ ^° theorem is proved. □ 
Proof of lemma 3. Let Ci = maxi<i<jvf /ii(0) and C2 > be such that 
hi{C2) < Si/2 for all i. Then inequality (21) implies that 

(24) L24Y^+^) - L2,.(y/) < Vxsi + CiI{L24Yl) < C2) a.s., 

where V^t^. = + ei/2. 

Let ft < t be the last time k before t when L2,i{Y/') < C2 (we let z^t = 
if such k does not exist). Then 

t 

L2,.(F/+i)-L2,.(y,°) < + 

< C1+C2+ max V Vife 
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For each x & V, let X* be a stationary sequence satisfying conditions (6)-(7). 
Then 



1 

— max 

t o<j<t 



t 1 * 1 * 

k=j k=j 



Here the first summand on the RHS tends to both a.s. and in mean, since 
the sequence {ip^^. .} is stationary ergodic with negative mean E-^^^ . < 

—£i/2. The second summand also tends to in mean uniformly in x & V, 
due to uniform integrability. Indeed, for any A > let i? be such that 



E 



i/j^ jldV'x i\ ^ ^) — ^5 X G X. Then for any x £ V and any k 



E 



< A + R6k 



and a similar inequality holds for ip^^. .• Therefore, 



< 2A + 2i?-^(5fc. 



t 



k 

X'',i 







As A is arbitrary small and 6t 0, the second summand on the RHS of the 
latter inequality may be made arbitrarily small, and the result follows. □ 



3. Stability of two systems with two random multiple-access 
protocols and a finite number of stations. In order to improve perfor- 
mance and usability of laptops, tablet computers and other portable equip- 
ment, such devices are nowadays often endowed with several wireless inter- 
faces. For example, if a device is equipped with the WiFi interface (pro- 
tocol standard IEEE 802.11) and the WiMax interface (protocol standard 
IEEE 802.16), the user has a wide range of possibilities for the wireless 
exchange of data. However, it is known that if the same device uses both 
these standards, their data streams interfere with each other and the speed 
of transmission and throughputs therefore deteriorate. There are currently 
attempts to minimise the effect of such interference (see, e.g. [2], [25]). 

In this paper we introduce a simplified model that mimics the essential 
features of both standards and of the interference between them described in 
the previous paragraph. The IEEE 802.11 standard uses a random-access al- 
gorithm for message transmissions, therefore multiple message transmissions 
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may be attempted simultaneously and collide with each other. Such conflicts 
are then resolved according to a specific algorithm. In the IEEE 802.16 stan- 
dard message transmission follow a schedule and no conflicts occur. In our 
model, we assume that in the random-access algorithm every station trans- 
mits with a certain probability, independently of everything else (cf. ALOHA 
algorithm, see [1] and [21]). Each message that experienced a collision then 
simply returns to its origin and is treated as any other message. In other 
words, no conflict-resolution mechanism is implemented. Our model also as- 
sumes that the scheduling is completely symmetric: every node receives time 
intervals of the same duration to transmit their messages, successively. 

We consider two cases. In the first case the system has the so-called MAC- 
coordinator (see [2] and [25]) which gives a priority to the IEEE 802.16 
protocol. This case is considered in subsection 3.1. The second case concerns 
a system without a MAC-coordinator and is considered in subsection 3.2. 

3.1. Network with a MAC-coordinator. Assume there are M identical 
stations numbered 1, ...,M. There are 2 types of messages called "red" and 
"green" , and each station has two infinite buffers where these messages may 
be stored, one for each type. 

We make a few assumptions: 

Assumption 1. Time is slotted, stations may only start transmissions at 
the beginning of a time slot, and each transmission time is equal to the length 
of a slot. Hence, we may assume that the events (such as arrival of a new 
message, the beginning of a transmission and the end of a transmission) may 

only happen at time instants 1,2, We also assume that the transmission 

channel is such that, at a given time slot, at most one red and at most one 
green message may be transmitted. Note also that any single station cannot 
transmit two messages (red and green) simultaneously during the same time 
slot. Summarising, in any time slot, there may be no transmissions at all. 
Otherwise, there may be a transmission of either only one red message or 
only one green message or one red and one green message, but in the latter 
case the transmissions have to be made by different stations. 

Assumption 2. Transmissions of red messages are scheduled and do 
not collide. More precisely, for t = 1,2,..., time slot t is scheduled for 
a transmission from node i{t) = {{t — 1) mod M) -|- 1: if the queue of red 
messages at that node is non-empty, then there is a (successful) transmission 
of the first one of them; otherwise there is no transmission of red messages 
at that time slot. 

Assumption 3. Transmissions of green messages follow the well-known 
ALOHA protocol (see [21]): at a given time slot, every node that is not 
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transmitting a red message and whose queue of green messages is not empty, 
transmits a green message (say, first in the queue) with probabiHty p, inde- 
pendently of everything else. Then one of three events may occur: 

• Only one node attempts to transmit a green message. Then the trans- 
mission is successful; 

• No transmission attempted; 

• Two or more nodes attempt transmissions of green messages. Then all 
these transmissions fail due to collision, and the messages stay in their 
queues. 

Assumption 4. Red messages arrive in the system as an i.i.d. sequence 
{^*}t>o with a finite rate Xr = E,^* (here ^* is the total number of new 
red messages within time slot {t — l,t]). Similarly, green messages arrive 
independently as a renewal sequence {r]^}t>o with a finite rate Xq = Ery*. 
Every arriving message is assigned to a node at random with equal proba- 
bilities 1/M. 

Let -R* and G* be the numbers of red and green messages respectively 
in the queue of node i at the beginning of a time slot t. The sequence 
{ {Ri , . . . , R\j ) } forms a Markov Chain, and so does the sequence {{R\, R\,j , G* , . . . , G\j ) } . 
Note also that the latter Markov Chain describes the state of the system 
completely. We will say that the system is stable if its underlying Markov 
Chain is positive recurrent. 

For such an algorithm we prove the following 

Theorem 3. Assume Xr < 1. Then the system is stable if 
(25) 

Xg<{1-Xr)p, if M=l, 

XG<XR{M-l)p{l-pf'-^ + {l-XR)Mp{l-p)^'-\ if M>1 

and unstable if the opposite strict inequality holds. 

Remark 2. The conditions of the theorem seem to be intuitively clear. 
Consider a single time slot. The average number of new green messages in 
a time slot is equal to Xq. Let us now consider the average number of green 
messages leaving the system. For that, assume that all nodes have a green 
message (this may be thought as the "worst-case" scenario). Assume also 
that the Markov chain representing the states of the queues of red messages 
in all nodes is in its stationary regime. In that case, when we look at a single 
time slot, the queue of red messages of the node that is scheduled to transmit 
a red message in the time slot will be non-empty with probability Xr (this 
is quite clear intuitively but see Appendix B for a proof) and empty with 
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probability 1 — \fj. In the former case, there are M — 1 nodes that can 
successfully transmit a green message and probability of success for each of 
them is p{l — p)^^~'^ and therefore the expected number of green messages 
leaving the system is (M — — p)^-'^"^. The latter case may be considered 
in a similar way. 

The simple heuristic argument presented above depends heavily on the 
symmetry of the system. The symmetry is also essential for our strict anal- 
ysis. 

Proof of Theorem 3. We give a proof only for M > 1. The case M = 1 
may be considered following the same lines and applying straightforward 
changes. We start by proving stability, a proof of instability follows. 

Proof of stability. We apply Theorem 2. Denote by ^* and ??* the 
numbers of new red and, respectively, green packets arriving at time slot t 
to station i. Since the total number of red arrivals in time slot t is ^* and 
each of them chooses one of the stations at random, each ^* has a condi- 
tional binomial distribution with parameters ^* and 1/M and Xli^^i ~ 
Similarly, for t = 0, 1, . . ., a random variable has a binomial distribution 
with parameters 77* and 1/M. Denote also by 



a* 



1, with probability p, 
0, with probability 1 — p 



the sequence of i.i.d. random variables representing the decisions taken by 
the nodes on whether or not to attempt a transmission of a green mes- 
sage. Then the Markov Chain {{R\, R\^.j,Gi, ...,G\j)} has the following 
transitions: 



\Ri + Cl-I{Rl>0}, if i = i{t), 



where, as before, i{t) = {{t-1) mod M) + l. Further, let 7* = a*I{G* > 0}. 
Then 



t+i 



G* + r/*-7* n (1 - 7*) (1 - 7*(i)I{i?*(„) = 0}) , if i / 



G* + r?*-I{i?H0}7in l-7h if i = i{t). 



Introduce now a new Markov Chain {{Ri, R\,j,G\, G\,f)} where the 
first M components (representing the states of the red queues) are the same 
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as before, and the remaining components (representing the states of the 
green queues) have the following transitions: 



t+i 



Here 7* = a^IIG^j ^ ^J'' ^'^'^ 

In words, the Markov Chain {{R\, R'y,G\, ...,G\.j)} represents the 
state of the system where each station with an empty green queue (if not 
blocked by a transmission of a red message) may send (and does so with 
probability p) a "dummy" packet which interferes with (dummy or legiti- 
mate) packets of other stations. 

Since 7* < a* a.s. for all i and t, it follows from the two systems of 
equations displayed above that the new Markov Chain dominates the initial 
one: if {(G}, Gl,)} = {{Gl, ...,G\,)}, then 

{{R{, R\f, G{, ...,G\.j)} < {{R{, R\,f,G{, G\.f)} 

a.s., for any t. Hence, to prove stability of the initial Markov Chain, it is 
sufficient to prove stability of the new one. As it follows from Theorem 2, 
Lemma 3 and the symmetry of the system, it is sufficient to show that condi- 
tion Bl and conditions of Lemma 3 hold for the sequence {{Ri, Rm, 
Note that it forms a Markov chain in this case. For simplicity, we will 
also omit the coordinate index 1 in all the functions appearing in con- 
ditions. Take function L2{y) = y and consider the state of the Markov 
Chain {{R\, R\^,G\)} after every M steps. 
One can see that, for t = 0, 1, . . ., 

. M M 
(26) - Gf = (X^ +^ - I = 0) 11(1 - af +^) 

^1=1 j=2 
M / M 

-X;af+Ml(i2f+^ = 0)n(l-af+^) + l(i?f+^>0) J] U " "f 

j=2 \ k=2 k>2,kj^j 

Let Sj^^ = X]fcf=i ^l^'^'^ be the total number of arrivals into the red queue 
of node i within the consecutive M time slots, and let 
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M M 

^iT = E ^l''^' - 1 = 0) n(i - "f -^^) 

i=l j=2 
M / M 

-5]af+^' i(x,+5f^=o)n(i-«i)+i(^.+^.>o) n 

i=2 y fc=2 k>2,kjtj 

Then the RHS of equation (26) may be estimated from above by random 
variable 9?^/ 1 ^ • 

We prove now that condition Bl and conditions of Lemma 3 hold for the 
Markov Chain {Ri, R\^j, G\} in M steps. Indeed, we have 

E(|Gf+i \{R\,...,RIj,G\) = {n,...,rM,gi)) < Emax|^7?i,M| , 

since at most "^f^i r]\ new messages may arrive in the system and at most M 
messages may leave the system. Condition Bl thus holds. Conditions (i) 
and (ii) of Lemma 3 clearly hold for random variables Take Ci = 

C2 = M, then condition (iii) of Lemma 3 also holds with hi{yi) = Cil{L2^i{yi) 
C2). To verify the last condition (iv), we note that P(-R*(j) = 0) ^ 1 — Xr 
as t — 7> 00 (this follows from a known general result that, for a stationary 
Markov chain Zt+i = max{Zt — l,0)+Xt with i.i.d. integer- valued increments 
{Xt} such that Exi = c < 1 and P(xi ^0) = 1, we have with necessity 
P{Zt = 0) = 1 — c, in order to keep our paper self-contained, we provide a 
proof in Appendix B). Therefore, we get 



J EClMdx) = {1-Xr)p{1-p) 



M-l 



M 

+ 5^ ((1 - Xr)p{1 - pf'-' + Xrp{1 - p)''-^) < 0, 
i=2 

provided the conditions of the Theorem hold. □ 
Proof of instability. As was mentioned in the stability proof, P(i?*^^j = 
0) — )• (1 — Xr) as n — )■ 00. We can choose t so large that \P{R^-^^^ = 
0) — (1 — Xr)\ < 5 for an arbitrarily small 6 > 0. For simplicity of cacu- 
lations, let us assume that P(i?*j-j^ = 0) = (1 — Xr) (it will not be difficult 
for the reader to repeat the same proof with an extra 6 added and then let 5 
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go to 0). We prove that, for any i = 1, . . . , M , ^ oo with at least a hnear 
speed, i.e. 

(27) liminfG*/t>0 a.s. 

(note that by the SLLN the speed of growth can not be superhnear). Con- 
sider the embedded epochs M, 2M, . . . , kM, . . . that are the multiples of 
M. Choose a positive number >> 1. Since all states in the positive M- 
dimensional lattice are communicating, there exists an a.s. finite (random) 
time, say, T G {kM, k > 0} such that Gf > N for all i. Starting from time 
T, all coordinates of the process G^'^'^ coincide with those of the auxiliary 
process G'^^^ which starts with the same G^ = G^ - until the first time 
when one of the coordinates becomes zero. Since, for any i, the increments 
Q{k+i)M _ QkM ^Qj-T^ stationary ergodic sequence with a positive mean, 
say A (which is a difference of the RHS and the LHS in equation (25)), 

Gf'^/k^A a.s. 

and, for any e > 0, one can choose iV >> 1 such that inf/>o > 
M -\- 1 with probability at least 1 — e. If one takes e < 1/M, then all the 
coordinates of G^^'^ always stay above M after time T with probability at 
least 1 — Me > 0. Then the same holds for the coordinates of the process 
G'''^'^. Since G*"*"^ - G\ > -I a.s., it then follows that, with probability at 
least 1 — Me > 0, the coordinates of G* stay strictly positive for all t > T. 
Since e may be taken as small as possible, (27) follows. 

□ 

Recall that i{t) = {{t — 1) mod M) + 1. Introduce a Markov Chain 

{{R\,.. .,Rm) ,t>0} = {(-R*(t),^*(i+i),^i(t+2)' • • • >^i(t+M-l)) > o} 

and another Markov Chain {R, G) = {{R\, . . . , R^j, G\,..., G\,^) , t > O}, 
where a similar interchange of the G-coordinates is also made. In words, the 
new Markov chain is obtained from the original one with a cyclic change of 
coordinates such that at every time slot the first coordinate corresponds to 
the node whose first red message is scheduled for service (if there is any). 

Corollary 2. Under the assumptions of Theorem 3, the Markov Chain 
{R, G) is Harris ergodic. 

Proof. Indeed, the new chain is aperiodic and positive recurrent. The 
latter follows from the fact that the Lyapunov function used in the proof of 
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Theorem 3 is the sum of ah coordinates and hence does not change when the 
order of the coordinates is changed. To infer ergodicity it is then sufficient 
to show that the state (0, 0) is achievable from any other state. To show 
that, note first that there exists a compact set V which is positively recurrent 
for the new chain. This implies that with a positive probability the chain 
will reach a state (ri, ■.,rM-,gi, ■■gAi) £ V- Due to compactness of V, there 
exist finite and such that X^flj^ ri < and X^fl^ gi < g^ for all points 
from V. It is then clear that there exists a finite time such that with a 
positive probability all red and all green messages will be transmitted and 
no new messages will arrive. Hence the state (0, ...,0) may be reached in a 
finite number of steps from any other state. 

□ 

3.2. Network without a MAC- coordinator. Now consider the system which 
differs from the system described above only in the following: we assume here 
that a station which is transmitting a red message can also attempt a trans- 
mission of a green message and, if that happens, these two transmissions 
collide. 

Now the following may happen at a time slot t regarding red messages: 

• There is one attempted (and successful) transmission. This happens 
when the queue of the red messages of node i{t) is non-empty and there 
is no attempted transmission of a green message by the same station 
(this may happen if either the queue of green messages is empty, or it 
is not empty but the station takes a decision not to transmit a green 
message); 

• There is one attempted (and unsuccessful) transmission. This happens 
when both queues of red and green messages of station i[t) are non- 
empty and the station decides to attempt a transmission of a green 
message. 

• There are no attempted transmissions. This happens when the queue 
of the red messages of node i{t) is empty; 

The following may occur regarding green messages: 

• There is one transmission attempt. Then it is successful, if it did not 
originate from node i{t) or if it did originate from node i{t) but its 
queue of red messages is empty; 

• No transmission attempted; 

• Two or more transmissions attempted. In this case all these transmis- 
sions fail due to the collision. 

We are still assuming that the ALOHA algorithm is used to govern the 
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behaviour of the green queues of all stations. For this system, the following 
holds. 

Theorem 4. The system is stable if \r < 1 — p and 
(28) 

Ag < (l - T^) P, if M = 1, 

Ag<^(M-1)p(1-p)*'^-i + (i-^)mp(1-p)^^-i, if M>1. 

Proof of Theorem 4 may be given following the lines of the proof of 
Theorem 3. The only difference is that one needs to bound the Markov 
Chain representing the state of the system under consideration by a Markov 
Chain representing the state of the system where each station with an empty 
green queue transmits a "dummy" message with probability p. 

Note that in this case the first component (counting the number of red 
messages) of the initial Markov Chain describing the state of the system 
is not a Markov Chain itself, so Theorem 2 can not be applied directly. 
However, the Markov Chain used to bound the initial one from above has 
the first component which is a Markov Chain itself, and the use of Theorem 2 
is justified. 

Acknowledgement. The authors would like to thank Maxim Grankin for 
his simulation work that numerically confirmed the correctness of stability 
results in the case of a network without a MAC-coordinator. 

Appendix A. Proof of (6)-(7). Assume p < 1 (where p is from the 
minorisation condition, see Assumption ^3)~ otherwise the result is obvious. 
Consider, for simplicity, the case m = 1 only (the general case requires an 
extra technical work which is not essential). For x € y, consider the standard 
splitting identity 

P.(x'..)-P.(-) + (i-P) ''-'^\'^-'-'"'" 

l-p 

and denote the fraction in the RHS by Qx(') which is a probability measure. 
We know that sigma-algebra Bx is countably generated. Therefore, one can 
define two measurable functions f,g:Xx\f^,\\^X such that if ?7 is a 
r.v. uniformly distributed in [0,1], then f{x,U) has distribution Px{-), for 
X G X, and g{x,U) has distribution Qxi'), for x G V - see, e.g., [15] for 
background. 

Introduce three sequences of mutually independent r.v.'s, each of which 
is i.i.d.: 

1) a sequence of — 1-valued r.v.'s with common distribution P(/3t = 
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1) =p = i-P(A = o), 

2) a sequence {Ut} of uniformly distributed in [0, 1] r.v.'s, and 

3) a sequence {Wt} of rf- valued r.v.'s with common distribution //. 

Then Markov chain X* may be represented as a stochastic recursive se- 
quence (SRS): 
(29) 

= {PtWt + (1 - ^t)9(.x\ Ut)) ev) + f{x\ Ut)i{x' ex\v). 

The pairs {X^,l3t) also form a time-homogeneous Markov chain. Start with 
X° = x eV. Let To = and, for /c > 1, 

Tk = mm{t > Tk-i : X* G F}. 

Further, let 

9 = mm{k > : /Jt,, = !}• 

Then 6 has a geometric distribution with parameter p, F{9 = k) = p{l—p)'', 
k = 0,1, . . .. Let K = Tg + 1. Note that r.v. X'^ has distribution n and that, 
for X G V, 

Ea;K < soE6' + 1 =: C 

where sq is from (3). Clearly, C does not depend on x & V. Then, in partic- 
ular, random variables k are uniformly bounded in probability: 

sup Pa;(K > t) — 0, t — oo, 

by Markov inequality. 

Let now {/3t},{Ut}, and {Wt} be three other i.i.d. sequences which do 
not depend on the first three sequences. Consider a stationary Markov 
chain {X^} which is defined as follows: X^ has distribution tt and does 
not depend on all r.v.'s defined earlier, and 
(30) 

= {^tWt + (1 - ^t)9(.X\ Ut)) I{X' eV) + f{X\ Ut)l{X' ^X\V). 

Due to independence of the two SRS's, r.v. X*^ has distribution vr. Finally, 
let 

7 = min{j > : X^+J' G A, X^'^^ G A, p^+j = 1}. 

By aperiodicity, 7 is finite a.s. Also, it does not depend on k and, therefore, 
its distribution does not depend on x. Then one can define another sequence 
X" by 

X* = X%t < K + 7) + X%t > K + 7) 
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and find that, first, since k + 7 is a stopping time, {X*} and {X^} have 
the same distribution and, in particular, {X^} is also a stationary Markov 
chain (see, e.g. [23, p. 34] or [16]), second, random variables = k + 7 are 
uniformly bounded in probability, i.e. (7) holds, and, third, r.v.'s i' satisfy 
(6). □ 
Remark. The first intention of the authors was to find this result in [23]. 
However, Hermann Thorisson has confirmed that it is not there, but he is 
thinking to have it in the second edition of the book (with a complete proof, 
for any m > 1). 

Appendix B. 

Lemma 4. Consider a stationary Markov chain Zt+i = max(Z( — 1,0) + 
Xt with a stationary and ergodic sequence of integer-valued increments {xt} 
such that Exi = c < 1 and P{xi > 0) = 1. Then P{Zt = 0) = 1 - c. 

Proof. 

Let us rewrite the evolution equation as 

Zt+i = Zt-l + I{Zt = 0} + Xt, 

where I{- } is the indicator function. With a positive integer T we now have 

T-l T-l 

Zt+T = Zt-T+Y, HZt+^ = 0} + ^ xt+i- 

i=0 i=0 

If we now divide both sides of the latter expression by T, take expectations 
and let T tend to infinity, we will get 

lim ^±tI = -l+P = +c, 

T-i-oo T 

where we used the Strong Law of Large Numbers and the stationarity of the 
Markov chain {Zt}. Using stationarity again, we conclude that the LHS of 
the latter equality is equal to 0, and the proof is complete. □ 
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